SCALAR CURVATURE RIGIDITY WITH A VOLUME 

CONSTRAINT 



PENGZI MIAQi AND LUEN-FAI TAM^ 

Abstract. Motivated by Brendle-Marques-Neves' counterexam- 
ple to the Min-Oo's conjecture, we prove a volume constrained 
scalar curvature rigidity theorem which applies to the hemisphere. 



1. Introduction 

Recently, Brendle, Marques and Neves [6] have solved the long- 
standing Min-Oo's conjecture [15] by constructing a counterexample. 

Theorem 1.1 (Brendle, Marques and Neves P). Suppose n > 3. Let 
g be the standard metric on the hemisphere §" . There exists a smooth 
metric g on S", which can be made to be arbitrarily close to g in the 
C°° -topology, satisfying 

• the scalar curvature of g is at least that of g at each point in S" 

• g and g agree in a neighborhood o/5S", 
but g is not isometric to g. 

In this paper, we observe that if the metric g in Theorem 11.11 is 
assumed to satisfy an additional volume constraint, then it must be 
isometric to g. Precisely, we have 

Theorem 1.2. Let g be the standard metric on S" . Let g be another 
metric on S" with the properties 

• R{g) > R{g) m §^ 

• H{g) > H{g) on dS^ 

• g and g induce the same metric on dS"^ 

where R{g), R{g) are the scalar curvature of g, g, and H{g), H{g) are 
the mean curvature ofT, in {^,g), {^,g)- Suppose in addition 

Vig) > Vig), 
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where V{g), Vijj) are the volume ofg, g. If \ \g — g\\c'^(n) is sufficiently 
small, then there is a diffeomorphism (f : Q ^ Q with (f\j: = id, the 
identify map on S, such that (p*{g) = g. 

Theorem 11.21 is indeed a special case of a more general result: 

Theorem 1.3. Let {^l,g) be an n-dimensional compact Riemannian 
manifold, of constant sectional curvature 1, with smooth boundary S. 
Suppose II + H'j > (i.ell + H'y is positive semi-definite), where 7 is 
the induced metric on S and II, H are the second fundamental form, 
the mean curvature ofTj in {^l,g). Suppose the first nonzero Neumann 
eigenvalue /i of {fl, g) satisfies fi > n ~ 

Consider a nearby metric g on Q with the properties 

• R{g) > "^in — 1) where R{g) is the scalar curvature of g 

• H{g) > H where H{g) is the mean curvature of S in {Q, g) 

• g and g induce the same metric on S 

• y{.9) > ^{9) where V{g), Vijj) are the volumes of g, g. 

If Wq ^ qWc^IP.) sufficiently small, then there is a diffeomorphism ip 
on Vt with v^ls = id, such that (p*{g) = g. 

As a by-product of the method used to derive Theorem 1 1.3 1 we obtain 
a volume estimate for metrics close to the Euclidean metric in terms of 
the scalar curvature. 

Theorem 1.4. Let Vt dW^ be a bounded domain with smooth boundary 
S. Suppose 11 + ^^7 > (i.e. 11 + ^^7 is positive definite), where II, H 
are the second fundamental form, the mean curvature of S in M" and 
7 is the metric on S induced from the Euclidean metric g. Let g be 
another metric on H satisfying 

• H{g) ^ H, where H{g) is the mean curvature ofH in {^,g) 

• g and g induce the same metric on S. 

Given any point a G M", there exists a constant A > — ^^^rfy— 
depending only on Q and a, such that if \\g — ^Ilc3(n) is sufficiently 
small, then 

(1.1) v{g)-V{g) > [ R{g)<i> dvolg 

where $(x) = — \ x — + A > on 

Theorem 11.41 may be compared to a previous theorem of Bartnik [2] , 
which estimates the total mass [Ij of an asymptotically flat metric that 
is a perturbation of the Euclidean metric. 
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Theorem 1.5 (Bartnik [2]). Let g be an asymptotically flat metric 
on M'^. If g is sufficiently close to the Euclidean metric g (in certain 
weighted Sobolev space), then 

(1.2) 167rm(^) > / R{g) dvolg 

where xn{g) is the total mass of g. 

Our proofs of Theorems 11.21 - [L4l follow a recent perturbation analysis 
of Brendle and Marques in [5] , where they established a scalar curvature 
rigidity theorem for "small" geodesic balls in S". 

Theorem 1.6 (Brendle and Marques 0). Let C 6e a geodesic 
ball of radius 6. Suppose 

(1.3) cosS>^2=. 

Let g be the standard metric on Let g be another metric on Q with 
the properties 

• R{g) ^ n{n — 1) at each point in Q 

• H{g) > H at each point on dQ 

• g and g induce the same metric on dQ 

where R{g) is the scalar curvature of g, and H{g), H are the mean 
curvature of dQ in {^l,g), {^,g)- If g — g is sufficiently small in the 
C'^-norm, then f*{g) = g for some diffeomorphism ip : Vt ^ Vt such 
that v^Iqq = id. 

In Theorem 11.61 the condition 01.31) is equivalently to 

(1.4) /7>4tan(5 

because the mean curvature H of dB{5) is (n — 1)^^- As another 
application of the formulas in Section ^ we obtain a generalization of 
Theorem 11.61 to convex domains in 

Theorem 1.7. Let Q C be a smooth domain contained in a geodesic 
ball B of radius less than |. Let g be the standard metric on Let 
11, H be the second fundamental form, the mean curvature of dVt in 
{^l,g). Suppose Q is convex, i.e. 11 > 0. At dfl, suppose 

(1.5) i7>4tanr 

where r is the g-distance to the center of B. Then the conclusion of 
Theorem \1.6[ holds on Q. 
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Theorem 11.71 is an immediate consequence of Theorem 15.11 in Section 
O In a simpler setting, where the background metric ^ is a flat metric, 
we have 

Theorem 1.8. Let Q be a compact manifold with smooth boundary S. 
Suppose there is a flat metric g onQ such that II+H'j > (i.e. 11+ 
is positive semi- definite), where II, H are the second fundamental form, 
the mean curvature of S, and 7 is the induced metric on S. Given 
another metric g on Q such that 

• Rig) > on n 

• H{g) >H atT. 

• g and g induce the same metric on E, 

\ \9 ~9\\c'^{n) sufficiently small, then f*{g) = g for some diffeomor- 
phism Lf -.Vl ^ VL with lp\y. = id. 

Similar calculation at the infinitesimal level provides examples of 
compact 3-manifolds of nonnegative scalar curvature whose boundary 
surface does not have positive Gaussian curvature but still has positive 
Brown- York mass [TJ [8]. We include this in the end of the paper to 
compare with known results in [T7]. 

Theorem 1.9. Let S C M" be a connected, closed hypersurface satis- 
fying II + > 0, where II, H are the second fundamental form, the 
mean curvature ofTj, and 7 is the induced metric on E. Let Q be the 
domain enclosed by E in M". Let h be any nontrivial (0,2) symmetric 
tensor on VL satisfying 

(1.6) diY-gh = 0, tT-gh = 0, /i|ts = 0. 

Let {g(t)}\t\<e be a 1-parameter family of metrics on Q satisfying 

(1.7) g{0)=g, g'{0) = h, R{g{t)) > 0, g{t)\Tj: = gWi:. 
Then 

(1.8) [ Hda-g > I H{g{t))da-g 

for small t ^ 0, where H{g{t)) is the mean curvature ofE in {Q,g(t)). 

This paper is organized as follows. In Section [21 we derive a basic for- 
mula concerning a perturbed metric (Theorem 12.11) . which corresponds 
to Theorem 10] of Brendle and Marques. In Section [31 we prove 
Theorem II. 3 [ which implies Theorem II. 2 [ In Section [H we give a proof 
of Theorem 11.41 In Section [5l we consider other applications of the 
formulas in Section [2] and prove Theorem 11.71 - 11.91 
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2. Basic formulas for a perturbed metric 

Let Q be an ra- dimensional, smooth, compact manifold with bound- 
ary S. Let ghe a. fixed smooth Riemannian metric on Q. Given a tensor 
7], let "I //I" denote the length of rj measured with respect to g. Denote 
the covariant derivative with respect to g by V. Indices of tensors are 
raised by g. Let Rikji denote the curvature tensor of g such that if g has 
constant sectional curvature k, then Rikji = n{gijgki — QiiQkj)- Consider 
a nearby Riemannian metric g = g -\- h where h is a symmetric (0, 2) 
tensor with \h\ very small, say \h\ < ^. 

The following pointwise estimates of the scalar curvature of g and 
the mean curvature of S were derived by Brendle and Marques in [5] . 

Proposition 2.1 (Brendle and Marques [^). The scalar curvatures 
R{g), R{g) of the metrics g, g satisfy 

\R{g) - R{g) + (Ric(^), h) - (Ric(^), h') 

+ - Ig'^g'^g'^VMpVih,, + \\V{tTM' 

+V,[g''g^'{Vkh,i-Vih,k)]\ 
< C {\h\\Vh\^ + \h\') 

where Ric((7) is the Ricci curvature of g, h"^ is the g-square of h, i.e. 
{h'^)ik = g''''hijhki, (■,■) is taken with respect to g, and C is a positive 
constant depending only on n. 

Remark 2.1. If the background metric g is Ricci flat, i.e Rik = 0, then 
there will be no \h\^ term in the above estimate. That is because 

R{g) = g^'R^k - g''g'' {V.,khji - V.,ih,k) + g^'g^'g,, (r?,r^, - V],Vl) , 

where each term on the right, except g^^Rik, involves derivatives of h. 

Proposition 2.2 (Brendle and Marques [5]). Assume that g and g 
induce the same metric on S, i.e. /i|te = where TS is the tangent 
bundle of T,. Then the mean curvatures H{g), H{g) of Tj in {Q,g), 
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{^,g), each with respect to the outward normals, satisfy 

2 [H{g) - H{-g)] - U(z7, V) - \h{V, vf + ^ h{e^, vf \ H{g) 



n-l 



< C{\h\'^\Vh\ + \h\^) 

where {ca \ l<a<n — l}isa local orthonormal frame on S, V is 
the g-unit outward normal vector to S, and C is a positive constant 
depending only on n. 

To derive the main formula f l2.23p in this section, we let 

(2.1) DR-g{h) = -Ag{ti-gh) + diVgdiY-gh - (Ric(^), h) 

be the linearization of the scalar curvature at g along h. Here "A^, 
divg" denote the Laplacian, the divergence with respect to g. 

Lemma 2.1. With the same notations in Proposition \2.1\ assume in 
addition div gh = 0, then 

R{g) - R{g) = DR-g{h) - \DRg{h^) + {h,v\igh) - ^ (|V/i|' + \V{iigh)\'') 

+ h'^h'%k,i + E{h)+%iElih)) 

where E{h) is a function and Ei{h) is a vector field on Q satisfying 

\E{h)\ < C{\h\\Vh\^ + \hf), \Ei{h)\ < C\h\^\Vh\ 
for a positive constant C depending only on n. 
Proof. First note that 

(2.2) - V. (Vfc/i,7 - Vih.k)] - (Ric(^), h) = DR-g{h). 

Suppose g'^ = g'^ + r'^. Then r'^ = —h''^ + Ef{h) where /i*^ = g'^hjig^^ 
and \E2{h)\ < C\h\'^. Hence, 

where \E^{h)\ < C|/ip. Therefore, 
(2_^) 

-v.[(^7V - t'g'')iykh,i - Vih.u)] 

=Vi[{t''y' + -gV - Ef'\h)){Vkh,i - Vih.k)] 

= ^l^g\h\^ + (/l, Vhl-g{h))g - d\YgdlM-g{h'') " ^'(Vfc/l,7 " ^ ih j ■ 
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(2.4) 

\g''t'9'"^ihkrNih,, =hlwgdw-g{h^) - ^(Ric(^), /i^) + hi'' h'^'RiUji- 
The lemma follows from Proposition EIH dMD, (Q and ([MD. □ 



Next, let DHg{h) denote the linearization of the mean curvature at 
g along h. Proposition 12.21 implies 
(2.5) 

n-l 

h{V,V)H{g) - (2VeXe„,I7) - V^/i(e 

a=l 



For later use, we note the following equivalent expression of DHg{h) 
(see [H], (34)] for instance) 

(2.6) DH-g{h) = ^ Mtr-gh) - div-gh]{v) - divsX} , 

where X is the vector field on S dual to the 1-form /i(l7, ■)|ts- 
Let DRU-) denote the formal (^-adjoint of DRg{-), i.e. 



(2.7) DRl{\) = -{AgX)g + V|A - ARic(^) 

where A is a function and V?A denotes the Hessian of A with respect 
to g. The content of the following lemma had been used in [13]. 

Lemma 2.2. Let p be any smooth (0, 2) symmetric tensor on Q, then 
I DRg{p)\ dYo\-g = I {DRl{X),p) dvolg - I 2DH-g{p)X da-g 

(2 8) 

+ j \y{il-g{p) -P{V,V)) da-g 

where Xy = djjX denotes the directional derivative of X along V. 
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Proof. Let Y be the vector field on S dual to the 1-form p(z/, ■)|te- 
Integrating by parts, one has 

(2.9) 



DR-g{p)X dYo\g - / {Dm{X),p) dvol 



-Xdj7{tTgp) + {tigp)du\ + Adivgp(z/) VA) dag 

I A[-(^(trgp) + divgp(z7)] - (r, V^A) da-g + I \y{ti -g{p) - p{V,V)) da-g 
Jn Jt, 

/ A[-(^(trgj9) + divgp(z7) + divsF] da-g + / Xtj {ii g{p) - p{V ,V)) da-g 
Jt, Jt, 



where V (■) denotes the gradient on S with respect to the induced 
metric. From this and (12. 6p the Lemma follows. □ 

Using Lemma [221 we can estimate ^^[R{g) — R(jj)]\ dvo\g. 

Proposition 2.3. Suppose g and g induce the same metric on S and 
h satisfies divgh = 0. Given any C"^ function A on Vt, one has 



[R{g) - R{g)] A rfvolg 



{KDRl{\)) d^o\-g-- 



{h\DR*JX)) dvolg 



+ 
+ 
+ 



iti-gh){h,Vl\) + ^-h'^h^'R,uji\ - -A\^h\' + \W{ti-gh)\')\ 



1 



civoL 



\x\ 



X-n dag 



2 

hnn{X,V A) dag 



1^ 



- (hnnYHig) - -n(X,X) - -\X\'H{g) 



X dag - / (2 - 2tTgh)DHg{h)X dag 
Jt 



2 2 2 

+ / E{h)XdYol-g- I E{{h)V^Xdvo\-g+ I Fi{h)X da-g 
Jn Jq Jt 

where II is the second fundamental form of S in {Q, g) with respect to 
V, X is the vector field on E that is dual to the 1-form h{en, ■)\tt, E{h) 
and E\{h) are as in Lemma lKlj and Fi{h) is a function on S satisfying 

\Fi{h)\ < C\h\^\Vh\ 

for a positive constant C depending only on n. 
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Proof. By (12 .Sp with p = h, using the fact that /i|t(s) = 0, we have 

(2.10) / DR-g{h)X dvol-g = / {DR*g{X), h) dvolg - / 2DHg{h)X da-g. 
Jn Jn is 

By the second hne in (12. 9p with p = h? , and integrating by parts, we 
also have 



/ ~DR-g{h^) + \{h,V^il-gh) dYOl-g 

Jn 2 

/ -l-{DR*{X),h^)+tr-gh{h,V''x) dvo\-g + B 
Jn 2 



(2.11) 

where 
(2.12) 

13 = j I [A^d^n - \h\'d^X - X{diY-gh'){i7) + {e){i7,VX)] da-g 

+ / [Xh{V,VtTgh) -tTghh{V,VX)]dag. 
Jy. 

To compute let {cq, | l<a<?T, — Ijbean orthonormal frame 
on S and let e„ = V. Denote V also by " ; ", thus hij.^ = ^k^ij. The 
assumptions /ilrs = and divgh = imply the following facts on S: 

(2.13) = {Kn)^ + 2\X\^ {hXr. = iKny+\X\^ {hX^ = K^Ka. 

(2.14) {h^V.VX) = [{K^f + |X|2]A;„ + V'^A), 

(2.15) h 



n-1 



(2.16) = (trg/i);„ - ^ = (tr§/;,);a - 2II(X, e^), 



n— 1 

(2.17) = (div/i)„ = + ^ = + hnaH{g) + II{X, e„), 

/3=1 



(2.18) 



n-l 



= {diVgh)n = hnn;n + ^ Ka;a = Kn;n + div^X + h^nH (g) , 



(2.19) 2Di7g(/i) = {tT-gh),n - divsX, 
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where (12191) follows from (EE]). By (12:T6D - ( 12181) . we have 
(2.20) 

— '^h h -\- h h — h h 

'Ji i'na"'nayn ~ ""nn"'nn;n "'na"'nn;a 

= - U{X,X) - 3H{g)\X\' - H{g){Knf - Kn^^iY^X - {X,VhT-,h). 
By (1212|) . (127[3|) . ( imj) . (12120]) and integration by parts, we have 
(2.21) 



B 



A. 



1 



Kn{x,y A) 
1 



n(X,X) - -H{g)\X\' - -H{g){KnY + 2KnDHg{h) 



Xdag. 



E{{h)V,\ rfvolg+ / \Fi{h) da. 



+ 

Note that 

(2.22) / (yiE{{h))\ d^o\g 



where \Fi{h) = {Ei{h),V)\ < C\h\^\Vh\. Proposition |23] now follows 
from Lemma ( CTIj) . dm]) . ( ETj) . and ([02]). □ 

The formula ( I2.23P next is a general form of [HI Theorem 10], which 
Brendle and Marques derived for geodesic balls in S". 

Theorem 2.1. Suppose g and g induce the same metric on S and h 
satisfies divgh = 0. Given any function A on Q, one has 

(2.23) 

/ [R{g) - R{g)] A dvol-g + / (2 - tr^/i) [//((?) - H{g)] A t^a^ 
{h,Dm{X)) dvol-g-- [ {h\DR*J\)) dvol-g 



dvoh 



+ 
+ 



{ti-gh){h,VlX) + ]-h''h^'lku,i\ - 7(|V/i|2 ^ |V(trg/i)nA 



--(/i„„)2iJ(^) - -{ll{X,X) + H{g)\X\ 



+ / A.„ 



A rfcTn 



/ {-l)hnn{X,V A) rfcTg 



+ / E{h)XdYo\-g+ / ZX/i)ViA c/volg + / F{h)Xda, 
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where E{h) is a function and Z{h) is a vector field on Vt satisfying 

\E{h)\ < C(|/i||V/ip + l/iH, \Z{h)\ < C\h\^\Vhl 
and F{h) is some function on S satisfying 

\F{h)\ < C{\h\^\Vh\ + \h\^). 
Proof. Proposition 12.21 implies 

(2.24) 2[H{g) - H{g)] = 2DHg{h) + J{h) + F^ih) 

where 



J{h) 



and F2{h) is some function on S satisfying \F2{h)\ < C(|/ip| V/i| + 
Therefore 



{2-h^n)[H{g)-H{g)] 



(2.25) 



(2 - 2hnn)DHg{h) + 



H{g) 



(I2.23P now follows readily from Proposition 12.31 and fl2.25p . □ 

The term DR^{X) in fl2.23p may suggest that one consider a back- 
ground metric g which admits a nontrivial function A such that DR^{X) = 
(such metrics are known as static metrics [ID]-) For instance, if g is 
the standard metric on S" and A = cosr, where r is the ^-distance to 
a point, then 02.230 reduces to the formula in Theorem 10]. 

Besides static metrics, one can also consider those metrics g with the 
property that there exists a function A such that 

(2.26) DRliX) = g. 

These metrics were studied by the authors in [i3\ and ^14j. In this case, 
the terms 

[ {h,DRl{X)) dvol-g - \ I {h\DRl{X)) dYo\-g 
Jn ^ Jo. 

in f l2.23p become 

tigh dYO\g I \h\^ dYO\g. 

To compensate these terms, one can include the difference between the 
volumes of g and g into f l2.23p . 
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Corollary 2.1. Suppose g is a metric on Vt with the property that there 
exists a function A satisfying DR*g{\) = g. Let g = g + h be a nearby 
metric such that g and g induce the same metric on S and h satisfies 
diVgh = 0. Let V{g), V{g) denote the volume of {Q,g), {Q,g). Then 

(2.27) 

- 2iV{g) - V{g)) + [ [R{g) - R{g)] A dvol^ + / (2 - tr^/i) [H{g) - H 



1 



--{\Vh\' + \Vs{tT,h)\')^ 



+ 



+ 



4 n - 1 

Y^—R{9){t'Cghf + {h,Ric{g)) {ti gh) + ^hijhkiRikji 



dvoh 



A dvoU 



-h^h^^fH{g) - i(II(X,X) + /7(^)|Xp) 



- -\X\ 



A dan 



da-g+ / (-l)/i„„(X,V A) da,^ 



+ / G{h) dvol-g + / E{h)\ dYol-g + / Z\h)V.,\ dYo\-g + / F{h)\ da-g 
Jn Jn Jn is 

where G{h) and E{h) are functions on Q satisfying 

\G{h)\ < C\hf, \Eih)\ < Ci\h\\Vh\^ + \h\'), 
Z{h) is a vector field on Q satisfying 

\Z{h)\ < C\h\^\Vh\, 
and F{h) is a function on S satisfying 

\F{h)\<C{\h\''\Vh\ + \h\'). 
Proof. The difference between the volumes of g and g = g + h is 



(2.28) Vig)-V{-g)= j \{iigh) + 



+ G{h) c/vol 



31 



where Giji) is a function satisfying |G(/i)| < C|/;.|^ for a constant C 
depending only on n. Suppose DR^yX) = g, i.e. 

-(AgA)^ + VjA-ARic(^)=^. 

Taking trace, one has A^A = j^[R{g)\ + n]. Thus, 

1 



(2.29) 



V?A 

V g/\ 



1 — n 



TOA + l]^ + ARic(^). 



flTTTI) follows from (E^SD, fl^:^ and (OHD. 



□ 
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3. VOLUME CONSTRAINED RIGIDITY 

We prove Theorem 11.31 in this section. First, we recall its statement: 

Theorem 3.1. Let {^,g) be an n- dimensional compact Riemannian 
manifold, of constant sectional curvature 1, with smooth boundary E. 
Suppose II + H'j > (i.ell + H'j is positive semi- definite), where 7 is 
the induced metric on S and II, H are the second fundamental form, 
the mean curvature ofH in {^l,g). Suppose the first nonzero Neumann 
eigenvalue ji of (fi, g) satisfies fi > n — 

Consider a nearby metric g on Q with the properties 

• -R(fi') ^ " 1) where R{g) is the scalar curvature of g 

• H{g) > H where H{g) is the mean curvature of S in {fl, g) 

• g and g induce the same metric on S 

• y{.9) > ^{9) where V{g), Vijj) are the volumes of g, g. 

^f\\9~9\\c'^{Q.) sufficiently small, then there is a diffeomorphism ip on 
with (p\y, = id, which is the identity map on T,, such that (f*{g) = g. 

Proof. Fix a real number p > n. By O Proposition 11], if | \9—g\\w^^p(n) 
is sufficiently small, there exists a W^'^ diffeomorphism (f on Q with 
V^|s = id such that h = ip*{g)—g is divergence free with respect to g, and 
1 1^1 lvK2.p{n) < N\\9 ~ 9\\w^'P(n) for some positive constant N depending 
only on {VL^g). Replacing g by f*{g), we may assume g = g + h with 
divgh = 0. We want to prove that if ||/i||ci(Q) is sufficiently small and 
g satisfies the conditions in the theorem, then h must be zero. 

Since q has constant sectional curvature 1, we choose A = — ^ such 

' n— 1 

that DR*g{X) = g. Corollary [O then shows 
(3.1) 

-2{V{g)-V{g))-^ [ [R{g) - R{-g)] dvol-, 

^—jj2-ti-gh)[H{g)-H{g)] da-, 

^^-i-^ ^ [-{n + l){tTghf + 2\h\^ + |V/ip + |V(trg/i)P] rfvolg 

+ Ji^^) ^ [^hnnfH{-g) + 2(n(X, X) + H{g)\X\^)\ da, 

-C\\h\\c.i^^)\f {\h\'' + \Vh\'')dYo\-g+ [ \h\Ua-, 
Un Jt, 

for a constant C depending only on (i^, (?). 



n 

> 
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Using the variational property of /z, we have 
(3.2) 



|V(trg/i)pdvolg > 



By (E2HD, Jn^rgh dvolg is related to {V{g) - V{g)) by 
(3.3) 



/ tTghdYolg = 2iVig)-Vig)y 
In 



In 



+ 2G(/i)| dYol-g, 



where G(/i) < C\h\^. 

Given any constant < e < 1, using (13. 2p and the fact > j^itigh)"^ 
and \Vh\^ > ^\V{tTgh)\^, we have 

(3.4) 

/ [-in+l){tT-gh)'' + 2\h\^+\Vh\^+\V-gitT-gh)\^] dVOl-g 

Jn 

2-el . r(l-e) 



> 



> 



e|/ip + e|V/ip + 
e|/i|2 + e|V/ip + 



(n + 1) + 



(tr^/i)' + 



+ 1 



n 



1N 2-e (1-e) 
-(n + 1) + + ^ -fi + fi 



n 



J V{-g) 



n n 

2 

ti ah d\o\? 



{il-ghf 



\y{iTgh)\' 

dvoXn 



dvo\-r 



Since /i > n — we can chose e (depending only on /x and n) such 



that 
(3.5) 



.^ 2-e (1-e) 
(n + 1) + + ^ ^/i + /i 



> 0. 



n n 

Then it follows from ([S3D, (IS3D and ([S3D that 
(3.6) 

/ (-(n + l)(trg/i)2 + 2|/i|2 + |V/i|2 + |V(trg/i))|2) rfvol^- 

>ef [\h\' + \Vhf) dYo\-g-C^{V{g)-V{g)f-Ci [ \h\Uag 
Jn Jn 

where Ci is a positive constant depending only on {^,g). 

At the boundary E, the assumption II+H{g)'y > imphes H{g) > 0, 
therefore 



(3.7) 



[{hnnfHig) + 2(II{X,X) + Hig)\X\^)] dcr-g > 
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for any h. By f l3.ip . (13. 6 p and (13. 7p . we have 

-S{n-l){V{g)-V{-g))-^ I [R{g) - Rig)] dYo\ 

Jn 

-4 / {2-tT-gh)[H{g)-H{g)] da-g 

JT. 

(3.8) > e / {\h\^ + \Vh\^) dYo\-g 

Jn 

-C{V{g)-V{g)f-C [ \h\' dvol^ 

Jn 

-C\\h\\c^^A [{\h\'+\Vh\')dYO\g+ [ \h\'dag 

Un Jy, 

for some positive constant C depending only on {VL^g). 
Finally, we note that 

(3.9) {V{g) - V{g)f <c(^j^ \h\ dvo\^ {y{g) - V{g)) 

by (13. 3 p and the assumption V{g) > V{g). Also, by the trace theorem, 

(3.10) \\h\\m'n)<C\\h\\w^,2^n) 

for a constant C only depending on f2. Therefore, by (13. Sp . (13. 9p . (I3.10p 
and the assumptions V{g) > V{g), R{g) > R{g) and H{g) > H{g) , 
we conclude that if is sufficiently small, then 

(3.11) > ^ / + iV/^n dvol-g 

2 Jn 

which implies h must be identically zero. This completes the proof. □ 



Remark 3.1. In Theorem 13. H if S is indeed empty, i.e {^,g) is a closed 
space form, its ffist nonzero Neumann eigenvalue satisfies fi > n as 
{Q, g) is covered by S". In this case. Theorem 13 . 1 1 say s that V{g) > V{g) 
implies g is isometric to g for a nearby metrics g with R{g) > R{g)- This 
could be compared to a more profound theorem known in 3-dimension: 
"// {M,g) is dosed 3-manifold with R{g) > 6, Ric{g) > g and V{g) > 
V{E>^), then {M,g) is isometric to S'^." (See [H Corollary 5.4] and earlier 
reference of [Sj [TT] ) 

When S 7^ 0, the boundary assumption 11 + H'y > in Theorem 13. II 
can be relaxed in certain circumstances. A detailed examination of the 
above proof shows, if 

(3.12) W{v,v) + > -(3j 



16 



Pengzi Miao and Luen-Fai Tarn 



for some positive constant /3, where /3 is sufficiently small comparing to 
the constant e in f l3.5p and the constant C in fl3.10p . then the conclusion 
of Theorem 13.11 still holds on such an (fi, (7). In particular, this shows 

Corollary 3.1. Let {M,g) be an n- dimensional Riemannian manifold 
of constant sectional curvature 1. Suppose Q G M is a bounded domain 
with smooth boundary E, satisfying the assumptions in Theorem \3.1\ i.e 
fJ' > n — and II + H'y > on S. Let Q G M be another bounded 
domain with smooth boundary S. If is sufficiently close to S in the 
norm, then the conclusion of Theorem \3. 1\ holds on Vt. 

It is known that the fist nonzero Neumann eigenvalue of S" is n (see 
[9l Theorem 3]). Therefore, Theorem 11.21 follows from Theorem 13.11 
Moreover, by Corollary 13. H Theorem 13. II holds on a geodesic ball in S"^ 
whose radius is slightly larger than |. 

By the next lemma, we know Theorem 13.11 also holds on any geodesic 
ball in S" that is strictly contained in 

Lemma 3.1. Let B{6) C &e a geodesic ball of radius 6. Let fi{6) be 
the first nonzero Neumann eigenvalue of B{6). 

(i) fi{6) is a strictly decreasing function of 6 on (0, |]. 
(n) For any < 5 < |, 

(sin 5)""^ cos 5 n 
J^{smt)-^dt (sin<5)2 

Proof. By [9, Theorem 2, p. 44], fi{6) is characterized by the fact that 

(3.13) {(sint)"-V}' + [fx{5) -{n- l)(sint)-='](sint)"' V = 
has a solution J = J(t) on [0, 6] satisfying 

(3.14) J(0) = 0, J'{6) = 0, f{t) ^0, V t G [0, (5). 

Given < 5i < ^2 < f , let Jj = Ji{t) be a solution to f l3.13p with 
fi{6) replaced by /i(5j), satisfying fl3.14p on [0, 6i], i = 1, 2. Replacing Jj 
by —Ji if necessary, we may assume that J- > on [0, 6i), hence Ji > 
on (0, 6i]. Define 



(sint)^ 



(sin t) 



By f l3.13p . fi satisfies 



f = -B i f2 

(sint)"-!^'' 
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Therefore, on (0,(5i], 

(3.15) (/1-/2)'- ^ ' ../.M...^.v«-i 



7--V^(/2 - ft) + [-"('^2) - ^(5i)](smt)' 



Note that /2(^) can be extended continuously to such that 

/i(0) = /2(0). Moreover, h > 0, h > on {0,6^), > = 

Let < to < ^1 be such that /i = /2 at to and /2 > fi for to < ^ < ^i- 
On (to,6i], one would have (/i — /2)' > if /i(52) > /^((^i)) which is a 
contradiction to /2 > /i- Therefore, /x((52) < This proves (i). 

To prove (ii), we further claim that to = 0, i.e. /2 > fi on (0,5i]. If 
not, there would be a nonpositive local minimum of (/2 — /i) at some 
to e (0,to]. At to, IKT5\i implies 

(3.16) = (/i - /2)' < [^(^2) - /x(5i)](sinto)"-^ < 

because < /2(to) < /i(to) and ^1(62) < /i(5i). Hence /2 > fi on (0, 61]. 
Integrating (13.151) on [0,(5i], we have 

(3.17) - /2(5i) = f\fi - f2)'dt > [/i(52) - /x(5i)] f'ismty^-'dt. 

Jo Jo 

Therefore 

/2(5l) 



(3.18) M'5i)>M'52)+ . ^ 

(sint)'^-irft 

Now let Si = S E (0, |) and S2 = vr/2. Applying the fact that /i(|) = n, 
J2 = sint, and 

/2 = (sint)""^ cost, 

we have 

(sin (5)""^ cos 5 
KS) >n+ \ \ 

/q (sm t)"-idt 

(3.19) ^ I (sm^)"-^cos^(5 

/„ cost(sint)"~^(it 



r-5 

n 



sin^ 5 

Therefore, (ii) is proved. □ 

4. A Volume estimate on domains in 
On M", the standard Euclidean metric g satisfies DR^{X) = g with 

(4.1) x{x) = - ^^J_^^ \x-a\^ + L 
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where | • | denotes the Euchdean length, a G M" is any fixed point 
and L is an arbitrary constant. In this section, we use this fact and 
Corollary 12.11 to prove Theorem 11.41 in the introduction. First we need 
some lemmas. 

Lemma 4.1. On a compact Riemannian manifold {^,g) with smooth 
boundary S, there exists a positive constant C depending only on (fi, (?) 
such that, for any Lipschitz function cf) onH, there is an extension of 
(f) to a Lipschitz function (p on Q such that 



(4.2) 



+ |V0n dvol-g < c 



+ IV 



daj^ 



where V, denote the gradient on Vt, S respectively. 



Proof. Let d{-, S) be the distance to S. Let 5 > be a small constant 
such that the tubular neighborhood U25 = {x G f2| rf(x, S) < 25} can 
be parametrized by F : S x [0,25) — f/25, with F{y,t) = expyitvijj)) 
where expy(-) is the exponential map at ?/ G S and v{y) is the inward 
unit normal at y. In U25, the metric g takes the form dt^ + a*, where 
{(j*}o<t<25 is a family of metrics on S. By choosing 5 sufficiently small, 
one can assume a* is equivalent to 0"° in the sense that \ < cr^'iy^v) <2 
for any tangent vector v with (T°(w, f ) = 1, V < t < 25. 

Let p = p{t) be a fixed smooth cut-off function on [0, 00) such that 
< p < 1, pit) = 1 for < t < 5 and p(t) = for t > §5. On 

U2S, consider the function (j){y,t) = (j){y)p{t). Since is identically zero 
outside Usg = {x E Q\ d{x, S) < cp can be viewed as an extension 

of (h on Q. For such an 6, one has 



(4.3) 



and 



(4.4) 



'dvolg < 



25 



da^ dt < C6 



'dan 



dvol-g < 2 / (|Vp|V^ + |V0| V) dvol-g 
n Ju2s 

25 



<C6 ' '-'2 

'E 



dag + 2 



|Vt (j)\'da' dt 



< C 



^dag + 



^dajs 



where denotes the gradient on (E, a*) and C is a positive constant 
depending only on (^2,(7). fl4.2p now follows from f l4.3p and f l4.4p . □ 
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Lemma 4.2. On a compact Riemannian manifold {^,g) with smooth 
boundary S, there exists a positive constant C depending only on (fi, (?) 
such that, for any smooth (0, 2) symmetric tensor h on Q, one has 
(4.5) 



h\'dYo\-g<C[ / \h\'da-g + \\h\\c2(n) / \h\'da-g+ / \h\\Vh\'dYol 



3 . 

Proof. On n, let = \h\2 . By lemma I4.H there exists a Lipschitz 
function on f2 such that = and 



+ |V0|2j dVOl-g < C 

Let Ai > be the first Dirichlet eigenvalue of {^,g), then 

dvoU 



da?) 



dvolg < 2 



(4.6) 



<2 (j)^ dvolg + 2X^^ / |V(0 



^(ivoL 



< C 



where 
(4.7) 



\V(f)\^dvo\g 



da-g+ / |V0|^rfvolg 



9 



V|/i|2|^c/vol§ < - / |/i||V/;.rrfvolg. 



To handle the boundary term | V ^pdcTg, given any constant e > 0, 
one considers 



(4i 



|V"(|/i|2 + e)i|2rfcrg 



(|/ip + e)4As(|/ip + e)3rfag 



where As denotes the Laplacian on S. Let {cq, | a = 1, . . . , — 1} be a 
local orthonormal frame on S and e„ be the outward unit normal to E. 
Let H be the mean curvature of S with respect to e„. Denote covariant 
differentiation i7 by " ; ". Let i, j run through {1, . . . ,n}. One has 



(4.9) 



a 

= ^ 2{hijhij.aa + h'^j-a) ~ ^ ^ 2hijh 



> - C\\h\y^n)\h\. 
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Therefore, 
(4.10) 

Ae(|/.P + 6)1 =\{\h\^ + e)-U,|/.p - + 6)-^|v"|/.|T 



> - C\\h\\c^^^{\h\' + e)-*\h\ - -m' + e)-t|Vl/i|^'2 



16 



(4.11 



It follows from (gS]) and KTO^ that 

/ \V''i\h\' + e)^'da,<C\\h\\c2^n) IW + e)^h\da, 
Jt, Jt, 

+ lljV^{\h\' + e)^'da,. 
Letting e — )■ 0, one has 

(4.12) / |v''|/i|tprfa5<C||/i|b.(o) / \h\'da,. 
Jt, Jt, 

(1431) now follows from (gl]), (gZD and (|4J2|) . □ 

We recall the statement of Theorem 11.41 and give its proof. 

Theorem 4.1. Let Q C M" be a bounded domain with smooth boundary 
Tj. Suppose 11 + H'j > (i.e. II + ^^7 is positive definite), where II, H 
are the second fundamental form, the mean curvature of E in and 
7 is the metric on E induced from the Euclidean metric g. Let g be 
another metric on satisfying 

• g and g induce the same metric on S. 

• H{g) > H, where H{g) is the mean curvature ofH in {^l,g). 

Given any point a E M", there exists a constant A > — ^^-i) — ' which 
depends only on Q and a, such that if \ \g — g\\c'-i{n) is sufficiently small, 
then 

(4.13) V{g) - V{g) > [ R{g)<!> rfvolg 

Jn 

where $ = — ., ,. \x — aP + A > on Q. 

4(n— 1) I I 

Proof. Fix a number p > n. By the proof of [5l Proposition 11], one 
knows if \\g — g\\w^'P{n) is sufficiently small, then there exists a W"^'"^ 
diffeomorphism cp : Q Q such that (p\y, = id, h = ip*{g) — g is 
divergence free with respect to g, and | |vy3.p(n) < N\\g — g\\w3.v(yi) for 
a positive constant depending only on (fi,(?). In what follows, we 
will work with (j)*{g). For convenience, we still denote (l)*{g) by g. 
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Given a G M", consider A(x) = ~ 2(n-i) 1-^ — + -^^ where L is a 
constant to be determined. First, we require L > 2(^n-i) ^^^g&n \l ~ ^l"^ 
so that A > on 0. Since A satisfies DR^{\) = g, Corollary 12. II shows 

(4.14) 

- 2{V{g) - V{g)) + [ R{g)X dvolg + / (2 - ti-gh) [H{g) - H] X da, 



< 



4 



-iV/ipA (ivolo + 



1 



—iKuYH - -(n(x, X) + H\x 



A dan 



- -\x\ 



da?, 



-l)hnn{X,V''X) da-g 



+ / G{h) rfvolg + / E{h)X dvo\-g + / Z\h)V^X dvol-g + / F{h)X da, 
Jq Jq Jq Js 

where \G{h)\ < C\h\^, \E{h)\ < C{\h\\Vh\^ +\h\'^), \Z{h)\ < C\h\^\Vh\, 
\F{h)\ < C(|/ip|V/;-| + \h\'^) for some constant C depending only on Q. 

At S, A;„ and V A are determined solely by and a (in particular 
they are independent on L). Apply the assumption 11 + ^^7 > (which 
implies H > 0) and the fact \h\'^ = {hnnY + 2|Xp, we have 



~{K^fH-]^{ll{X,X) + H\X\ 



(4.15) 



+ A.„ 



- -\x\ 



+ {-l)hnn{X,W^X) 



<- LCi\h\^ + C2\h\ 



where Ci, C2 are positive constants depending only on VL and a. We 
fix L such that 

(4.16) LCi - C2 > 

and let m = I min^ A (note that A is fixed now). (I4.14l) - fl4.16l) imply 
(4.17) 

- 2{y{g) - V{g)) + / R{g)X d^ioX-g + /" (2 - ti-gh) [H{g) - H] X da-g 



< -m \VhY dYo\g-{LCi-C2) / \h\'da, 



CA / (|/i||V/i|2 + |/i|3)rfvolg 



'■\Vh\ + \h\^) da-, 
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where C3 depends only on Q, a and L. Apply Lemma [4.21 to the term 
J^\hf dvolg on the right side of (14. 171) . we have 



2iVig) - Vm 



/ R{g)X dvolg + / (2 - tigh) [H{g) - H] X da-g 



< 



m 



IV/iP dvoh - {LCi - C2) I \h\'^d(T-g 

s 

/iP da-, 



+ C\\h\\c.^^A / \Vh\^dYO\g + 

\Jn Js 
where C is independent on h. From this, we conclude that if ||'i||c2(f^) 
is sufficiently small, then fl4.13l) holds with $ = |A. This completes the 
proof. □ 

Remark 4.1. When C is a ball of radius R, one can take a to be 
the center of Q. In this case, by computing H, II and A;„ explicitly in 
( ]4.16p . the constant L can be chosen to be any constant satisfying 

1 4 



L > 



+ 



R' 



_2(n-l) {n-iy_ 

Remark 4.2. By the results in fl2[ [T7] based on the positive mass the- 
orem lini HH], a metric g on Q satisfying the boundary conditions in 
Theorem 14.11 must be isometric to the Euclidean metric if R{g) > 0. 
Therefore, a nontrivial metric g in Theorem 14 . 1 1 necessarily has negative 
scalar curvature somewhere. For such a g, Theorem 14.11 shows if the 
weighted integral J^R{g)^ dvolg is nonnegative, then V{g) > V{g). 

5. Other related results 

In this section, we collect some other by-products of the formulas 
derived in Section 2. First, we discuss a scalar curvature rigidity result 
for general domains in S". 

Theorem 5.1. Let Q G S"' be a smooth domain contained in a geodesic 
ball B of radius less than |. Let g be the standard metric on Let 
II, H be the second fundamental form, the mean curvature of'L = dQ 
in {Q, g) with respect to the outward unit normal V. Suppose II > —cy, 
where c > is a function on S and 7 is the induced metric on S. Let 
q be the center of B. Suppose atT, \ {q}, 

5 cos 9 + \J cos^ Q 



(5.1) 



H-c> 



tanr 



where r is the g-distance to q and 9 is the angle between v and Vr. 
Then the conclusion of Theorem \1.6[ holds on Q. 
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Proof. As before, replacing g by <f*{g) for some diffeomorphism (f, we 
may assume divgh = where h = g — g. On Q, let A = cos r > 0, where 
r is the (^-distance to q. At S \ {q}, we have 



(5.2) A;„ = — sinr cos6', |V"A| = sinr sin^. 

Apply Theorem I2.H using the fact DR^{X) = and the assumptions 
on R{g) and H{g), we have 

(5.3) 



\{\Vh\' + \V{tT,h)\') + ^{\h\' + {tr.hr) 



cosr dvoln 



< 



-^{Kr^fH - ^(II{X,X) + H\X\') 



cosr da?, 



+ 



S\{q} 

Cll/'-llcicn) 



(M' + ^I^P 



(sin r COS ^) (i(Tg + / | /i„ri||X| (sin r sin 6') (icig 



(|/i|^ + |V/ir) dvo\g+ / da. 



< 



— I /inn 11-^ I (sinr sin 



-(// — c) cosr — sin 



inrcos^^ {hnn? + ^{{H 



c cosr — smr cos 



0) 



dan 



+ C\\h\\c^n) I j^i\h\' + \Vh\') dvo\g + \h\' da-g 

for some positive constant C independent on h. 
Note that the assumption (15. ip implies 



- {H — c) cos r — (sin r cos ^) > 



(5.4) 
and 

(5.5) (iJ — c) cosr — (sinr cos6') > 0. 

By dSHD, dED and ([53]), we have 



(5.6) 



0< (-(// — c) cosr — sinr cos 6* ) (/i^ 



|/i„n| |X| (sinr sin ( 



+ - ((if — c) COS r — sin r cos i 



for any and X. The result now follows from (15. 3 p and (15. 6p . □ 



Remark 5.1. It is clear from the proof of Theorem 15.11 that the center 
q of B does not need to be inside Q. 
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Theorem 15.11 directly implies Theorem 11.71 in the introduction. 
Proof of Theorem |i. 7[ Choose c = in Theorem 15. 1[ Since 



4 > 



5 cos 6^ + Vcos2 e + 8 



for any 6, the result follows from Theorem 15.11 



□ 



Next, we consider a corresponding scalar curvature rigidity result 
when the background metric ^ is a flat metric. 

Theorem 5.2. Let Q be a compact manifold with smooth boundary S. 
Suppose g is a smooth Riemannian metric on Q such that g has zero 
sectional curvature and II + H'j > on S, where II, H are the second 
fundamental form, the mean curvature ofE, and'y is the induced metric 
on S. Suppose g is another metric on Q satisfying 

• R{g) > where R{g) is the scalar curvature of g 

• g and g induce the same metric on S 

• H{g) > H where H{g) is the mean curvature of S in {Q, g) . 
If \\g ~ 9\\c'^{n) sufficiently small, then there is a diffeomorphism 
on Vt with = id such that (p*{g) = g. 

where h = g — g. Choose 



Proof. As before, we may assume divgh 
A = 1 in fl2:23|) . one has 



h\vh\' + \v{tT,h)\' 



dvoln 



(5.7) 



+ 



ih^nYHig) + -{II{X,X) + H{g)\X\ 



dan 



< 



/ E{h) dvo\-g + / F{h) da-g 



where \F{h)\ < C{\h\'^\Vh\ + \hf) and \E{h)\ < C\h\\Vh\'^ by Remark 
12.11 The result follows from (15. 7p . □ 

To finish, we mention that the positive Gaussian curvature condition 
of the boundary surface in is not a necessary condition for the 
positivity of its Brown- York mass. 

Theorem 5.3. Let S C 6e a connected, closed hypersurface satis- 
fying II + H'j > 0, where II, H are the second fundamental form, the 
mean curvature ofT., and 7 is the induced metric on S. Let Q be the 
domain enclosed by S in M". Let h be any nontrivial (0,2) symmetric 
tensor on Q satisfying 

(5.8) diVgh = 0, tigh = 0, /i|ts = 0. 
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Let {g{t)}\t\<e be a l-parameter family of metrics on Q satisfying 

(5.9) g{0) = g, g'{0) = h, R{g{t)) > 0, g{t)\Tj: = ^|ts. 
Then 

(5.10) [ Hda-g > [ H{g{t)))da-g 



for small t 0, where H{g(t)) is the mean curvature ofT, in {Q,g(t)). 



[R{g{t)) - R{g)] dvolg - 2 [ [H - H{gm da, 

JT. 



t=0 



Proof. By Lemma [2 ■2[ one knows 

d_ 
di 

Direct calculation using Lemma [221 fl2.17p and (15. 8p shows 

(5.11) ^ 

^ [Rig{t)) - R{-g)] rfvol, ' ^ JjH - H{g{t))] da, 

= -11 m'dvol-g- I [{U{X,X) + H{g)\X\^)\ da-g 

which is negative by the assumption on II + H'y. Thus, for small t 

(5.12) 2 [[H- H{g{t))] da-g > [ [R{g{t)) - R{g)] dvol-g > 0. 



□ 



Given an h satisfying (11. 6p . a family of deformation {g{t)} satisfying 

4 

(ll.7p is given by g{t) = u{t)'^{g + th) for small t, where u{t) > is a 
conformal factor such that R{g{t)) = (see fl3{ Lemma 4]). 

An example of a non-convex surface S C M^, which is topologically a 
2-sphere and satisfies the condition II + Hj > 0, is given by a capsule- 
shaped surface with its middle slightly pinched. 
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